Prune Your Model Before Distill It

Knowledge distillation transfers the knowledge from a cumbersome teacher to small student. Recent results suggest that student-friendly is more appropriate distill since it provides transferrable knowledge. In this work, we propose novel framework, “prune, then distill,” prunes model first make and We provide several exploratory examples where pruned teaches better than original unpruned networks. further show theoretically plays role of regularizer in distillation, which reduces generalization error. Based on result, neural network compression scheme student formed based apply distill” strategy. The code available at https://github.com/ososos888/prune-then-distill.